Entailment knowledge base in natural language processing systems

ABSTRACT

Generating textual entailment pair by a natural language processing (NLP) system. The NLP system receives two input texts, such as a question and a candidate answer. The NLP system queries a database and retrieves passages likely to include text that support the candidate answer. The NLP system generates parse trees and performs term matching on the passages and scores them according to the matching. The NLP system detects anchor pairs in the question and in the passage and aligns subgraphs (within the parse trees) of one to the other based on matching. The NLP system identifies aligned terms in the question and the passage that are not in the aligned subgraphs. The NLP system identifies text fragments, for the question and the passage, within the non-aligned segments of their respective parse trees, that connect the aligned term to the aligned portion of the subgraph.

BACKGROUND

Embodiments of the invention generally relate to electronic naturallanguage processing systems, and more particularly to textual entailmentdetection.

Traditional computing systems generally have evolved to processstructured data. They do not, however, understand what the structureddata means, and how it relates to other data, unless this understandingis preprogrammed into the system's functionality. These computingsystems are even less capable of meaningfully processing unstructureddata. To the extent that they process unstructured data, the processingis rudimentary; there is little insight into what the data means.

Cognitive computing systems, on the other hand, generally derive insightfrom unstructured data. For example, a natural language processingsystem, such as n open domain question-answering (QA) system, receives anatural language input, often in the form of a question, attempts tounderstand the question beyond its immediately known properties, andreturns a corresponding answer. QA systems can provide thisfunctionality through several processing techniques and tools,individually or in combination. These techniques include, for example,information retrieval (IR), natural-language processing (NLP), knowledgerepresentation and reasoning (KR&R), machine learning (ML), andhuman-computer interfaces (HCIs).

One example of a QA system is DeepQA™, a system developed by IBM®Corporation of Armonk, N.Y., as described in This is Watson, Ferrucci etal., IBM Journal of Research and Development, Vol. 56, No. 3/4, May/July2012, incorporated herein by reference in its entirety (all trademarksreferenced herein are property of their respective owners). DeepQA'sarchitecture defines various stages of analysis in a processingpipeline. Each stage admits multiple implementations that can producealternative results. At each stage, alternatives may be pursuedindependently as part of a massively parallel computation. DeepQA may beimplemented based on an assumption that no one processing componentperfectly understands the question asked of the system. Rather, DeepQAmay generate many candidate answers by searching a variety of sources,on the basis of varying interpretations of the question, and of thequestion type(s). DeepQA may gather evidence in support of the candidateanswers, and evaluates them relative to one another, to arrive at afinal answer.

DeepQA may apply hundreds algorithms that analyze evidence alongdifferent dimensions, such as type classification (both question type,and lexical answer type), time, geography, popularity, passage support,source reliability, and semantics relatedness. This analysis produceshundreds of features with scores, each indicating the degree to which abit of evidence supports an answer according to one of these dimensions.Features for a candidate answer can be combined into a single scorerepresenting the probability of the answer being correct. DeepQA mayalso train statistical machine learning algorithms on prior sets ofquestions and answers to learn how best to weigh each of the hundreds offeatures relative to one another. The final results of the process maybe a ranked list of candidate answers, each with a final confidencescore representing the likelihood that the answer is correct, based onthe analysis of all of its supporting evidence.

In one implementation, DeepQA may be deployed using the UnstructuredInformation Management Architecture (UIMA) architecture. UIMA refers toa software system that provides an infrastructure for large-scaleanalysis of unstructured data to discover useful information. The UIMAarchitecture may include, in one implementation, a set of threeframeworks: the Java Framework, the C++ Framework, and the UIMA-ASScaleout Framework. The frameworks define a series of interfaces andmanage a variety of components and data flows between those components.The components may have different functionalities in analyzingunstructured data. For example, components analyzing unstructured textmay include a language identification component, a language-specificsegmentation component, a sentence boundary detection component, and anamed entity detection component (an entity may be, for example, aperson, a company or organization, a geographical place, etc.). Systemsusing the UIMA architecture may be deployed on a cluster of networkedcomputing nodes, with UIMA having the capability to wrap its componentsas network services scaled on large volumes by replicating processingpipelines over the cluster.

In a typical example, which will be referenced with respect toembodiments of the claimed invention, a DeepQA system, such as IBMWatson® (hereinafter, “Watson”), may receive a natural language questionthrough an HCl (a human-computer interface). For example, the user maytype in a natural language question into a query field in a web browser.Watson may perform one or more of the following general processing stepsbased on the received question: Question and Topic Analysis & QuestionDecomposition; Hypothesis Generation; Collecting & Scoring Evidence;Merging Evidence & Combining Confidence; Answer & ConfidenceCommunication. Each of these functions is briefly discussed below.

Question and Topic Analysis & Question Decomposition:

to understand the received question, DeepQA may determine what thequestion is asking for; the question's focus. In determining the focusof the question, DeepQA may identify the word or phrase that indicatesthe class of the thing the question is asking for, referred to as thequestion's lexical answer type, or “LAT”. Further processing may includefurther question analysis techniques, such as parsing (shallow, deep,syntactic, logical form, or other parsing), using for example, anEnglish Slot Grammar (ESG) parser and a predicate-argument structure(PAS) generator. ESG may produce a grammatical parse of a sentence andidentify parts of speech and syntactic roles such as subject, predicate,and object, as well as modification relations between sentencefragments. The PAS generator may be used to produce a more abstractrepresentation of a parse, suitable for other analytics processingfurther down the DeepQA processing pipeline.

Hypothesis Generation—Discovering Candidate Answers:

DeepQA may use data outputs from the question and topic analysis andquestion decomposition processes to generate multiple interpretations ofthe question, and to generate a variety of corresponding queries. Thesequeries can run against different structured and unstructured sourcesusing a variety of search mechanisms, with complementary strengths andweaknesses. Rather than attempt to directly answer the question at thispoint in the pipeline, DeepQA generates a broad set of candidateanswers. Each candidate answer, combined with the question, mayrepresent an independent hypothesis. Each hypothesis may become the rootof an independent process, in the pipeline, that attempts to discoverand evaluate supporting evidence in its candidate answer. For example,DeepQA may compute a metric, referred to as candidate binary recall,representing the percentage of questions for which the correct answer isgenerated as a candidate. This metric may reflect the goal of maximizingcandidate recall at the hypothesis generation phase in the DeepQApipeline.

Hypothesis Generation—Answer Types:

A class of evidence considered by DeepQA may be evidence of whether theanswer is of the right type. Type coercion is one technique that may beused. This technique takes a lexical answer type and poses a typehypothesis for each candidate. It then consults a wide variety ofstructured and unstructured resources using a diverse set of algorithmictechniques to gather evidence for, and against, the type hypothesis.Algorithms designed to use these resources independently quantify adegree of support for believing that the type hypothesis is true.

Collecting & Scoring Evidence:

As described, DeepQA may generate candidate answers and confidencescores that indicate the degree to which each candidate answer isconsidered to be an instance of the answer type. DeepQA may attempt tocollect and score additional evidence. Algorithms designed to performthese functions may be called evidence scorers. These evidence scorersmay generate a confidence score—a number that indicates the degree towhich a piece of evidence supports or refutes the correctness of acandidate answer. Multiple evidence scorers can work in parallel foreach candidate answer, and over different forms of evidence.

One type of evidence is passage evidence—paragraphs or segments of textfound in volumes of textual resources that might support the correctnessof a candidate answer. Passage evidence may be identified or selectedbased on a variety of techniques, such as grammar-based techniques andrelation extraction techniques. Grammar-based techniques addresssyntactic and shallow semantic features of language. They look for howthe words and structure of language may predict similarities in meaning.Relation extraction techniques look deeper into the intended meaning,attempting to find semantic relationships between concepts, although theconcepts may have been expressed with different words, or with differentgrammatical structures. Relation extraction may be performed using, forexample, manually crafted pattern specifications; statistical methodsfor pattern elicitation; or both.

Merging Evidence & Combining Confidence:

As an example, DeepQA may consider 100 candidate answers for somequestion. For each of these, DeepQA may find 100 pieces of evidence inthe form of paragraphs or facts from databases. Each evidence-answerpair may be scored by 100 independent scorers. Each scoring algorithmproduces a confidence. For any one candidate, there may be on the orderof 10,000 confidence scores—on the order of 1 million in total for asingle question.

At some point in the DeepQA pipeline, DeepQA may rank candidate answersaccording to their evidence scores, and judge the likelihood that eachcandidate answer is correct. This may be done using a statisticalmachine learning framework. This framework may be phase-based, providingcapabilities for manipulating the corresponding data and applyingmachine learning in successive applications to deal with issues such asnormalization, training with sparse training data, and merging relatedor equivalent candidate answers.

Answer & Confidence Communication:

DeepQA may communicate, to the questioning user, a set of scored answersranked according to their corresponding combined confidence scores. Inthis manner, the user is provided with a meaningful answer to the user'snatural language question that goes beyond mere query generation andexecution, and addresses the meaning of the user's question byunderstanding the question, and its answers.

SUMMARY

According to an embodiment of the invention, a method for generating atextual entailment pair, by an electronic natural language processing(NLP) system, receives first and second texts, and queries a passagedatabase using the first and second texts. The method retrieves apassage from the passage database in response to the query, and selectsanchor pairs in the first text and in the retrieved passage. The methodgenerates an entailment pair based on the selected anchor pairs, theentailment pair including a pair of text fragments.

According to a further embodiment of the invention, an NLP systemincludes a processor and a tangible storage medium for storing programinstructions to execute a method for generating a textual entailmentpair. The NLP system receives first and second texts, and queries apassage database using the first and second texts. The NLP systemretrieves a passage from the passage database in response to the query,and selects anchor pairs in the first text and in the retrieved passage.The NLP system generates an entailment pair based on the selected anchorpairs, the entailment pair including a pair of text fragments.

According to yet another embodiment of the invention, a computer programproduct for generating a textual entailment pair. The computer programproduct includes a tangible storage medium storing instructions forexecution by a processor of an NLP system. The instructions cause theNLP system to receive first and second texts, and queries a passagedatabase using the first and second texts. The instructions furthercause the NLP system to retrieve a passage from the passage database inresponse to the query, and to select anchor pairs in the first text andin the retrieved passage. The instructions additionally cause the NLPsystem to generate an entailment pair based on the selected anchorpairs, the entailment pair including a pair of text fragments.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a block diagram of a natural language processing (NLP) system,according to an embodiment of the invention.

FIG. 2 is a diagram of illustrative parse trees for processing by theNLP of FIG. 1 , according to an embodiment of the invention.

FIG. 3 is a flowchart of a method for generating an entailment phrase,according to an embodiment of the invention.

FIG. 4 is a block diagram of a computing device for implementing the NLPsystem of FIG. 1 , according to an embodiment of the invention.

FIG. 5 is a block diagram of an illustrative cloud computingenvironment, according to an aspect of the invention.

FIG. 6 is a block diagram of functional layers of the illustrative cloudcomputing environment of FIG. 5 , according to an aspect of theinvention.

DETAILED DESCRIPTION

Natural language processing (NLP) systems, such as Question-answering(QA) systems, analyze natural language text based on text in a textcorpus. One example is answering natural language questions. The text inthe text corpus, and the likely answer to the question, is also innatural language form. Traditional NLP techniques that rely ontechniques such as keyword matching fail to identify the correct answerin instances where the language used in the question does not match thelanguage used in the answer, even though the answer may exist in thecorpus, although stated in different words.

A solution that addresses this mismatch, between the text in thequestion and the text in the answer, is to use textual entailment.Textual entailment refers to a directional relationship between textfragments, where at least one text fragment implies the other. Forexample, “the car burned down” implies “car is disabled”. This need notbe true in reverse. For example, “car is disabled” need not imply that“car burned down”. Both fragments refer to the same truth, although thefragments may use one or more different words. Textual entailmentbetween a given pair of text fragments defines an entailment pair (alsoreferred to as an entailment phrase). An entailment pair is a definedentailment relationship between one text fragment (such as a word orphrase) and another text fragment. For example, consider the question inEXAMPLE 1, below, presented in natural language form, and itscorresponding answer, also in natural language form, which may appear ina corpus. EXAMPLE 1 also includes a list of entailment pairs—words usedin the question, and corresponding terms in the answer, which may implyone another under a textual entailment model.

Example 1: Question/Answer Entailment

Question Answer “What is the hormonal cause of “Levels of hCG arehighest during mild elevation of thyroxine and the first three months ofpregnancy, decrease in TSH during the first and result in the drop inserum TSH trimester?” and a mild increase in serum free T4.” What[candidate answer] levels of hCG [question focus] during the firsttrimester during the first three months of pregnancy mild elevation ofthyroxine mild increase in serum free T4 decrease in TSH drop in serumTSH cause of result in

Words or phrases used in the entailment pairs in EXAMPLE 1 use one ormore different words, but in a given pair, each essentially conveys thesame truth. For example, “during the first trimester” is defined toentail “during the first three months of pregnancy.”

It should be noted that although embodiments of the invention are oftendescribed in connection with a QA system, the invention is not limitedto QA systems.

In an embodiment, entailment pairs may be tagged and stored, as follows:

<EVENT> is cause of <EVENT> ← <EVENT> results in <EVENT>  <PERSON> tookhome<AWARD> ← <PERSON> won <AWARD> <ENTITY> has increase in <ENTITY> ←<ENTITY> has elevation of <ENTITY>  <ENTITY> shows decrease in <ENTITY>←  <ENTITY> has drop in <ENTITY> <PERSON> presents with <FINDING> ← <PERSON> shows signs of <FINDING> <PERSON> tested positive for<DISEASE> ← <PERSON> contracted <DISEASE>

Referring to the example entailment pairs above; in the first instance,“is cause of” is paired with“results in”. Each is defined as a phraseappearing between two statements describing events. For example, thefollowing two sentences convey the same truth, according to the firstentailment pair in the above list: “Event A is the cause of Event B”conveys the same truth as “Event A results in Event B”. In the secondexample in the list, the sentence “Person A took home Award B” conveysthe same truth as the sentence “Person A won award B”, according to thesecond entailment pair.

Using entailment pairs to answer natural language questions becomes achallenge considering the number of such phrases that may appear inlarge text corpora, and variations between different domains to whichthe text corpora belong. This may be the case, for example, where thesame text may support multiple (and even competing or contradictory)entailment pairs, depending on the domain to which it belongs.Developing or obtaining a textual entailment source is time consuming,expensive, and limited by practical considerations.

Accordingly, embodiments of the invention generate textual entailmentdata. In one embodiment, a QA system may consult the textual entailmentdata to answer a natural language question using a natural languageanswer.

FIG. 1 is a block diagram of a computing environment 100, according toan embodiment of the invention. For illustrative purposes, computingenvironment 100 is described in relation to a QA system 102, text corpus120, and entailment-phrase knowledge base 130. It shall be apparent to aperson of ordinary skill in the art that, although FIG. 1 describes a QAsystem 102, embodiments of the invention may be practiced independentlyof a QA system. In an embodiment, computing environment 100 may be acloud computing environment, and QA system 102 may be a cloud computingnode, as described in connection with FIGS. 4-6 , below.

Referring now to FIG. 1 , each component in computing environment 100 isdescribed below.

QA system 102 may be an electronic question-answering system, such asthe DeepQA™ system developed by IBM® Corporation of Armonk, N.Y. QAsystem 102 may include one or more processing components (also calledpipelines) that include, for example, a passage retrieval component 104,a passage-scoring component 106, an anchor-pair-detection component 108,and an entailment pair extraction and scoring component 110 (“entailmentcomponent 110”). In an embodiment, this data may be accessed from a datasource other than a QA system.

QA system's 102 components are described below in conjunction with theFigures and EXAMPLE 2. This example includes an illustrativequestion-and-answer pair that QA system 102 may receive (by passageretrieval component 104 or another component), a corresponding querythat QA system 102 may generate, a passage returned based on the query(the passage may be one of many stored in text corpus 120), and anentailment pair that QA system 102 may generate based on analysis itperforms on the question-answer pair and the returned passage. Thisexample will be referenced from time to time throughout the remainder ofthe discussion that follows to illustrate functions of QA system 102generally, and its various components specifically.

Example 2: Illustrative Question-and-Answer Pair, Query, and EntailmentPair

Question “What is a common benign cause of congenitalhyperbilirubinemia?” Answer “Gilbert's Syndrome.” Query #combine[passage20:6] (cause common benign hyperbilirubinemia congenitalindirect gilberts syndrome) Returned Passage “In many patients,Gilbert's syndrome is commonly a benign explanation for congenitalhyperbilirubinemia.” Resulting Entailment Pair <X> is cause of <Y> ← <X>is explanation for <Y>

Passage Retrieval Component 104

Passage retrieval component 104 generally receives a question-and-answerpair from an input source (the input source may be a user, system,process, or a combination thereof; in one instance, it may be a QApipeline), constructs an information retrieval (IR) query using thequestion and its identified correct answer, and retrieves a ranked listof passages from text corpus 120. Text corpus 120 includes textual data,selected for storage in text corpus 120, based on an expected degree ofrelevance of that textual data to a domain of the question. For example,if the question relates to the healthcare domain, the textual data maybe selected based on its expected degree of relevance to healthcare.While embodiments of the invention do not require that the textual dataincluded in text corpus 120 have any minimum degree of relevance to thequestion's domain, the higher the relevance, the more reliable theresulting analysis may be.

Passage retrieval component 104 may provide an output to one or moreother components in QA system 102, including passage-scoring component106.

Referring to EXAMPLE 2, above, passage retrieval component 104 mayreceive the question “What is a common benign cause of congenitalhyperbilirubinemia?” and the answer “Gilbert's Syndrome” as an input;generate the query “#combine [passage20:6] (cause common benignhyperbilirubinemia congenital indirect gilberts syndrome)”; and retrievethe passage “In many patients, Gilbert's syndrome is commonly a benignexplanation for congenital hyperbilirubinemia.” In this example,“passage20:6” is part of the query instruction to the passage retrievalengine that tells the retrieval engine what size of passages toretrieve; about 20 word passages by including words in 6 word increments(until sentence boundary is reached).

Passage-Scoring Component 106

Passage-scoring component 106 generally receives an output of passageretrieval component 104, for example the question-and-answer pair andthe returned passage in EXAMPLE 2. Functions of Passage-scoringcomponent 106 include term matching.

Given two passages of text (or a question and a passage), term matchingdetermines if pairs of terms in the two passages refer to the samereal-world entity. For instance, the term matcher can detect that theterm George Washington in the first passage and the term PresidentWashington in the second passage both refer to the same person and henceare “matched” by the term matcher. Similarly, the term matcher wouldmatch a term like back pain in the first passage with the term backachein the other passage. The term matcher may (but not necessarily)exhaustively evaluate all pair terms in two passages (one term from onequestion/passage, and the second term from the other passage), anddetermines if they match.

In an embodiment, term matching may include applying a variety of termmatchers to a pair of terms to determine if they are a match. Eachindividual term matcher may use a corresponding algorithm to decide iftwo terms match. One term matcher may decide that two terms are a matchif they have the same lexical string (string match). However, termmatchers may also use other external resources to decide if two stringsmatch. For instance, one matcher uses a lexical resource such asWordNet® to determine if two terms are synonyms of one another.Synonymous terms may be considered term matches by this matcher. Anothermatcher may exploit Wikipedia “redirect” information to decide when twoentities are the same (e.g., George Washington, GW and PresidentWashington all may redirect to the same Wikipedia page). In the medicaldomain, a resource such as UMLS may be used to get evidence of termmatching. Matching of terms across a pair of passages may then beachieved by applying a collection of term matchers and combining theirdecisions to identify term matches. Combining decisions from multiplematchers is done through either hand-coded rules, or through a trainedmachine learning classifier.

In addition to the above term matching, for a question and passage pair,where the passage contains one or more candidate answers to thequestion, QA system 102 may also have an additional custom term matchercalled the “question-candidate” term matcher. This special purpose termmatcher matches the “focus term” in the question (usually the wh-word orphrase) with the candidate answer in the passage. Take the followingquestion passage pair as an example:

-   -   Q: Which US president came to office during the World War II?    -   P: Harry Truman took the US presidency when the country was        still engaged in WWII.

Assuming “Harry Truman” is a candidate answer, this matcher will matchthe phrase “Which US president” (the focus of the question) with thecandidate term “Harry Truman” in the passage.

Passage-scoring component 106 generates representations (“graphs” or“parse trees,” which are used interchangeably here) of syntactic andsemantic relationships between terms in pairs of a question and aretrieved passage. Passage-scoring component 106 generates a graph forthe question, and a graph(s) for one or more of the retrieved passages.In one embodiment, passage-scoring component 106 may generate a graphfor the question, and graphs for all retrieved passages. Based on thegenerated graphs, passage-scoring component 106 aligns a focus of thequestion to an answer string in the passage under consideration, in aprocess referred to as focus-candidate alignment. This may be a subsetof the term matching functions that passage-scoring component 106performs.

Focus-Candidate Alignment—

In the focus-candidate alignment process, a question's focus may referto the part of the question that, if replaced by the answer, makes thequestion a standalone statement. For example, in the question “What drughas been shown to relieve the symptoms of ADD with relatively few sideeffects?”, the focus is “drug”, since if this word were replaced withthe answer—for example, “Adderall”— to generate the sentence “Adderallhas been shown to relieve the symptoms of ADD with relatively few sideeffects”, the resulting sentence would be a standalone statement. Thefocus often, but not always, contains the question's Lexical Answer Type(LAT).

A question may have more than one focus, and more than one portion ofthe question may be evaluated as a focus candidate. Focus candidates maybe generated and evaluated using focus generation and scoringalgorithms.

In the focus-candidate alignment process for the question, and thecorresponding passage under consideration, QA system 102 may identifyone or more strings in the passage for alignment to the focus term(s) inthe question. This may include the passage-scoring component 106generating one or more graphs for the question, and the passage underconsideration. Passage-scoring component 106 may also identify focussubgraphs for the question, and for the passage under consideration. Afocus subgraph for the question may refer, in an embodiment, to aportion of a question's graph, including connecting edges, that includesthe focus term, and all question terms such that the question term is(a) aligned to some passage term, and (b) has an edge connecting theterm to the focus term, or to a term in the connected subgraph. In graph202, for example, the focus term is “What”, which aligns with “Gilbert'ssyndrome”, and “is” is directly connected to “What”, and is aligned to“is” in the passage. No other terms directly connected to these twoterms are aligned to the passage. Therefore, the focus subgraph in thisexample is subgraph 202A: “What”→“is”.

Passage-scoring component 106 may align the subgraphs it identifiesbased on similarities between nodes in the subgraphs. The alignment maybe based on, for example, identical or similar nodes at the same nodelevel; two nodes being identified as equivalent using known entailmentpairs; or other criteria. How well two subgraphs align may be the basisfor scoring the alignment, which may form the basis for scoring theentailment pair generated by other components of QA system 102.

For each passage analyzed, passage-scoring component 106 may score thepassage based on the alignment scores of the focus subgraph. In oneembodiment, passage-scoring component 106 may do so by using a decayingsum of term alignment scores in the focus subgraph. The decaying sum isa weighted sum of term alignment scores, where the weights are inverselyproportional to the distance of the terms from the candidate term offocus term in the focus subgraph.

FIG. 2 illustrates two illustrative graphs and corresponding subgraphsfor the question and passage in EXAMPLE 2, above. Referring now to FIGS.1 and 2 , and EXAMPLE 2, passage-scoring component 106 generates aquestion graph 202 and a passage graph 204. Passage-scoring component106 identifies “what” as a focus term in question graph 202, based ondetermining that replacing “what” with “Gilbert's syndrome” in theanswer would generate the following standalone statement: “Gilbert'ssyndrome is a cause of benign congenital hyperbilirubinemia”.

With continued reference to FIGS. 1 and 2 , and EXAMPLE 2,passage-scoring component 106 identifies question subgraph 202A(is→what) and passage subgraph 204A (is→Gilbert's syndrome), and alignstheir terms, and their connecting edge(s). Specifically, passage-scoringcomponent 106 aligns “is” to “is”, and “what” to “Gilbert's syndrome”,and their respective connecting edges. While other subgraphs may beidentified here, there is no subgraph with aligned terms that connects“Gilbert's syndrome” to “congenital hyperbilirubinemia”. However, sincethe question and the passage are assumed to convey the same truth, QAsystem 102 may assume that there is an undetected alignment, which wouldhave been detected had there been an entailment pair defining therelationship between “Gilbert's syndrome” and “congenitalhyperbilirubinemia”.

Anchor-Pair-Detection Component 108

Anchor-pair-detection component 108 generally receives an output ofpassage-scoring component 106, for example question graph 202 (includingaligned subgraph), and passage graph 204 (including aligned subgraph204A) in EXAMPLE 2. Anchor-pair-detection component 108 may select pairsof anchor terms from each graph (in an embodiment, for each pair, oneanchor term may be selected from each of two corresponding subgraphs).Anchor-pair-detection component 108 may output pairs of anchor terms forprocessing by other components of QA system 102.

Anchor-pair-detection component 108 generally may select pairs of anchorterms from the question and the passage under consideration. An anchorterm generally refers to a term representing an entity that canplausibly be “linked” across passages or questions. Anchor terms mayinclude, for example, terms representing people, organizations,locations, temporal expressions, etc. Additionally, terms belonging tomany semantic categories representing physical entities (such asanimals, vehicles, buildings, etc.) also may be considered anchor terms.

In an embodiment, to detect anchor terms in a given piece of text, thetext may be pre-processed using a Named Entity Detector and a SemanticClass Tagger. Named Entity Detector is an NLP technology that canidentify “named entities”—persons, organizations, geopolitical entities(such as countries, cities, etc.)—within the specified text. Similarly,Semantic Class Taggers are NLP technologies that identify the semanticclass of terms in the input text. These NLP technologies may be builtupon machine learning models trained over human annotated text corpora.Some examples of open source and commercial of Named Entity Detectorsavailable in academia and industry: Stanford CoreNLP, Illinois NamedEntity Tagger, Alchemy API Entity Extraction. Examples of Semantic ClassTaggers include: Stanford Pattern-based Information Extraction andDiagnostics (SPIED) and Utah Bootstrapped Semantic Class Tagger (allmarks are property of their respective owners).

In one embodiment, for a given question and passage, a set of anchorpairs is identified as all pairs of terms that meet all of the followingcriteria: (1) one term is from the question, and one term is from thepassage; (2) the terms are both anchor terms; and (3) the terms havebeen matched to each other by the term matcher (for example, using theterm alignment functions performed by passage-scoring component 106).

In an embodiment, anchor-pair-detection component 108 may select a pairof anchor terms from question graph 202 and passage graph 204, byselecting one anchor term from question subgraph 202A (for example,“what”) and one anchor term from question subgraph 204A (for example,“Gilbert's Syndrome”). Similarly, Anchor-pair-detection component 108may select a pair of anchor terms from question graph 202 and passagegraph 204, by selecting one anchor term from nodes in graph 202 but notin subgraph 202A (for example, “congenital hyperbilirubinemia”), and oneanchor term from nodes in graph 204 but not in subgraph 204A (forexample, “congenital hyperbilirubinemia”). The anchor pairs are, in thisexample:

Pair 1: (“What”, “congenital hyperbilirubinemia”);

Pair 2: (“Gilbert's syndrome”, “congenital hyperbilirubinemia”).

Entailment-Pair-Extraction Component 110

Entailment-pair-extraction component 110 generally receives an input ofpairs of anchor terms generated by anchor-pair-detection component 108.Entailment pair extraction component identifies text fragments thatconnect the anchor terms in the question and the passage, and evaluatesthem as candidate entailment pairs.

In an embodiment, entailment-pair-extraction component 110 operates bytaking advantage of a likelihood that anchor terms in aligned subgraphsof the question and of the answer can serve as hints as to how terms intheir non-aligned subgraphs are related, i.e., non-aligned subgraphslikely contain entailment pair candidates. Accordingly,entailment-pair-extraction component 110 extracts entailment pairs fromthe question and passage graphs using the detected anchor pairs, andscores the extracted entailment pairs. QA system 102 may repeat thisprocess (including additional processes described above in connectionwith other components) to process the question in relation to additionalrelated passages returned by passage retrieval 104, to assign entailmentpair scores to candidates generated by entailment-pair-extractioncomponent 110.

Extracting an entailment pair may include, in an embodiment, identifyingtext fragments in a graph/subgraph of the question, and text fragmentsin a graph/subgraph of the passage under consideration, which connectthe anchor terms in the question and the passage (for example, subgraphs202B and 204B). These text fragments may be, for example, nodes in thegraphs/subgraphs. In an embodiment, the text fragments may include allnodes connecting the aligned anchor terms on the shortest path to thealigned focus subgraph.

A first fragment, extracted from the question, and a second fragment,extracted from the passage under consideration, define an entailmentpair (also referred to as an entailment phrase). The entailment pair maybe scored according to various scoring criteria, over successiveoperations of entailment-pair-extraction component 110, to score andrank the entailment pair.

Scoring functions may vary according to embodiments of the invention. Inone embodiment, a given extracted entailment pair may be assigned ascore based on the number of times it is extracted byentailment-pair-extraction component 110 for various passages underconsideration. For example, if an entailment pair is extracted from ahundred passages, it may receive a higher score than if it had beenextracted fifty times.

In an embodiment, the score assigned to an entailment pair may reflectthe average rank of the passages from which it is extracted, relative toother passages.

In an embodiment, the score assigned to an entailment pair may reflectthe average passage score of the passages from which it is extracted,(for example, the passages' rank may be as determined by passage scoringcomponent 106).

In an embodiment, the score assigned to an entailment pair may reflectthe number of different questions whose processing by QA system 102results in the entailment pair's extraction.

In an embodiment, entailment-pair-extraction component 110 may add oneor more of entailment pairs that it extracts to entailment pairknowledge base 130. In an embodiment, entailment-pair-extractioncomponent 110 may add only those entailment pairs whose score meetsselection criteria, such as a threshold value.

Referring now to FIGS. 1 and 2 , and EXAMPLE 2, above,entailment-pair-extraction component 110 may receive two pairs of anchorterms from anchor-pair-detection component 108, as described above:(“What”, “congenital hyperbilirubinemia”) and (“Gilbert's syndrome”,“congenital hyperbilirubinemia”). Entailment-pair-extraction component110 acts on the insight that these two anchor pairs are connected by acommon anchor term, “congenital hyperbilirubinemia”, and that theirrespective source texts (the question, and the passage underconsideration) likely convey the same underlying truth (the likelihoodmay be measured via a score). Entailment-pair-extraction component 110additionally acts on the insight that the non-aligned subgraphs of thequestion and the passage under consideration are likely to contain anentailment pair, which describes the relationship between terms in thenon-aligned subgraphs.

In the case of the non-aligned subgraphs in FIG. 2 ,entailment-pair-extraction component 110 hypothesizes that the phrase“is cause of” entails “is explanation for”, and generates the entailmentpair “<X> is cause of <Y>←<X> is explanation for <Y>”.Entailment-pair-extraction component 110 may assign an initial score of(0.01) to this entailment pair based on extracting it for the firsttime.

Entailment-pair-extraction component 110 may adjust the score as partof, or through additional, analysis functions. For example,entailment-pair-extraction component 110 may add (0.01) to this initialscore for each additional time that it extracts this same entailmentpair.

As a further example, entailment-pair-extraction component 110 maymaintain a list of passage ranks for each passage whose analysisresulted in extracting the entailment pair (for example, bypassage-scoring component 106), and weigh the score accordingly. Forinstance, as entailment-pair-extraction component 110 extracts theentailment pair from a passage, entailment-pair-extraction component 110may apply a weighting multiplier to the score of (0.01) to account for ahigh, neutral, or low passage rank.

As a further example, entailment-pair-extraction component 110 maymaintain a list of passage scores for passages from whichentailment-pair-extraction component 110 has extracted the entailmentpair, and apply a weighting multiplier to the score of (0.01), where themultiplier is determined based on an average passage score.

As a further example, entailment-pair-extraction component 110 may add(0.01) to the score for each question whose analysis results inentailment-pair-extraction component 110 extracting the entailment pair.

Referring now to embodiments of the invention in general, and FIG. 1 ,QA system 102 may use scoring techniques other than those describedabove, without departing from the spirit or scope of the disclosedinvention. Additionally, although descriptions of QA system 102 refer topairs of anchor terms, more than two anchor terms may be used.Additionally, entailment pairs may use more than two phrases.Additionally, entailment pairs (or more than two) may be extracted frompairs of retrieved passages, rather than a question and passage pair.Additionally, QA system 102 may perform a combination of thesetechniques.

With continued reference to embodiments of the invention, in general,the entailment generation functions described above need not beperformed using a QA system. Rather, these functions may be implementedindependently from a QA system, and may be used for purposes other thananswering questions.

FIG. 3 is a flowchart of a method 300 for generating a textualentailment pair by an electronic natural language processing system,according to an embodiment of the invention. Steps of method 300 may beimplemented using programming instructions executable by a processor inthe natural language processing system. In one example, steps of method300 may be executed by a processor in QA system 102 (FIG. 1 ).

Referring now to FIGS. 1-3 , steps of method 300 will be described inconnection with QA system 102 (FIG. 1 ) and graph 202 and graph 204(parse trees shown in FIG. 2 ), according to an embodiment of theinventions.

QA system 102 may receive a pair of texts, which may be referred to asfirst and second texts (step 302). In one embodiment, the pair of textsmay be a question-and-answer (QA) pair: the first text is in the form ofa question, and the second text is in the form of a candidate answer.The QA pair may be, for example, the QA pair in EXAMPLE 2 above(Question: “What is a common benign cause of congenitalhyperbilirubinemia?”; Answer: “Gilbert's Syndrome.”). Other embodimentsmay include different text pairs.

Passage-retrieval component 104 generates a search phrase based on thefirst and second texts and queries a passage database (step 306). Forexample, passage-retrieval component 104 may query text corpus 102 usingthe query string in EXAMPLE 2, above (Query: “#combine [passage20:6](cause common benign hyperbilirubinemia congenital indirect gilbertssyndrome”).

Passage-retrieval component 104 retrieves a passage from the passagedatabase, in response to the query (step 310). For example,passage-retrieval component 104 retrieves the passage in EXAMPLE 2,above (Passage: “In many patients, Gilbert's syndrome is commonly abenign explanation for congenital hyperbilirubinemia.”).

Passage-scoring component 106 may perform term matching between terms inthe first text and terms in the retrieved passage (step 314). Termmatching may be performed based on a term matching criteria. In oneembodiment, the term matching criteria may be to match a focus term inthe first text to a term in the retrieved passage. Passage-scoringcomponent 106 may score the retrieved passage, relative to the question,based on the term matching, using scoring criteria. The scoring criteriamay include, for example, a confidence value associated with thematching; the number of matched terms; the frequency with whichdifferent matching algorithms result in the same match; and othercriteria.

Passage-scoring component 106 may, in an embodiment, align a textfragment in the first text with a text fragment in the retrievedpassage, based on the term matching. The alignment may be based on thepassage score. For example, in one embodiment, the alignment between thetwo text fragments may be performed based on passages whosecorresponding alignment scores meet a minimum threshold value.

Passage-scoring component 106 matches at least one term in the firsttext with at least one term in the retrieved passage. Passage-scoringcomponent 106 may perform this function by generating one or more parsetrees of the first text and of the retrieved passage. Passage-scoringcomponent 106 identifies one or more focus terms in the respective parsetrees of the first text and the retrieved passage, and identifies one ormore focus subgraphs in the first text and one or more subgraphs in theretrieved passage for alignment to the focus subgraphs, based on theterm matching. Passage-scoring component 106 may align at least onefocus subgraph of the first text with a subgraph of the retrievedpassage. In one embodiment, the alignment may be based on how closelyfocus terms in the focus subgraph match terms in the retrieved passage.

In one embodiment, in the subgraph generation and alignment process, thefocus subgraph is a subgraph of the first text's parse tree thatincludes the focus term in the first text and its parent nodes.Similarly, the subgraph in the retrieved passage (to be aligned to thesubgraph of the first text), is a subgraph containing a term matched tothe focus term of the focus subgraph (the focus subgraph is a subgraphin the parse tree of the first text).

For example (with reference to EXAMPLE 2 and the graphs in FIG. 2 ),passage-scoring component 106 identifies “What” as a focus term in thequestion, and further identifies subgraph 202A as a focus subgraph ofgraph 202. Passage-scoring component 106 also identifies “congenitalhyperbilirubinemia” as a term in the retrieved passage that matches thefocus term “What” in the question, and further identifies subgraph 204Ain graph 204 as a subgraph of the matched term. Passage-scoringcomponent 106 identifies subgraph 202A and subgraph 204A as alignedsubgraphs.

Anchor-pair-detection component 108 selects two or more anchor termpairs in the first text and in the retrieved passage (step 318), where afirst anchor term in a given pair is from the first text and a secondanchor term in the pair is from the retrieved passage. The selection maybe based on detecting anchor terms in the first text and in theretrieved passage using term matching techniques, described above inconnection with passage-scoring component 106 (at step 314).

In one embodiment, anchor-pair-detection component 108 selects a firstanchor term in the first text matched by a term matcher to a secondanchor term in the first text. In one instance, the selection may bebased on matching the first anchor term to the second anchor term, wherethe first anchor term is in the focus subgraph of the first text, andthe second anchor term is in a subgraph in the retrieved passage alignedto the focus subgraph of the first text. In another instance, theselection may be based on matching a third anchor term, in the firsttext, to a fourth anchor term, in the retrieved passage, where the thirdanchor term appears in a subgraph other than the focus subgraph; thefourth anchor term appears, in the retrieved passage, in a subgraph notaligned to the focus subgraph.

In reference to EXAMPLE 2, above, anchor-pair-detection component 108identifies as anchor pairs, “What” in the question, and “congenitalhyperbilirubinemia” in the retrieved passage, as aligned term, where“What” appears in focus subgraph 202A, and “congenitalhyperbilirubinemia” appears in subgraph 204A, where the two subgraphsare aligned. Anchor-pair-detection component 108 also identifies asanchor pairs, “Gilbert's syndrome” in the non-aligned subgraph 202B inthe question, and “congenital hyperbilirubinemia” in the non-alignedsubgraph 204B.

Entailment-phrase-extraction component 110 may generate an entailmentpair (step 322) based on the anchor pairs selected/generated byanchor-pair-detection component 108. Entailment-phrase-extractioncomponent 110 generates the entailment pair by aligning a first textfragment, in the first text, with a second text fragment, in theretrieved passage, based on aligned first and second anchor terms in anfirst anchor pair, where the first anchor term appears in the first textfragment, and the second anchor term appears in the second textfragment. Entailment-phrase-extraction component 110 aligns a thirdanchor term in the first text with a fourth anchor term in the retrievedpassage, where the third anchor term is not in the first text fragmentand the fourth anchor term is not in the second text fragment.

Entailment-phrase-extraction component 110 identifies a third textfragment, in the first text, connecting the third anchor term to thefirst text fragment, and further identifies a fourth text fragment, inthe retrieved passage, connecting the fourth anchor term to the secondtext fragment.

In an embodiment, for a given text fragment among the third and fourthtext fragments, identifying the text fragment includes identifying in acorresponding parse tree (or graph) of a corresponding text (forexample, the first text or the retrieved passage), a shortest path (or asubgraph) connecting the anchor term in a non-aligned subgraph to a termin an aligned subgraph.

The third and fourth text fragments identified byentailment-phrase-extraction component 110 constitute an entailmentphrase.

Referring more specifically to EXAMPLE 2 and FIG. 2 ,entailment-phrase-extraction component 110 identifies subgraph 202A andsubgraph 204A as aligned subgraphs, and identifies “What” and “Gilbert'sSyndrome” as aligned anchor terms in those subgraphs (the alignmentitself may, but need not, be performed by another component).Entailment-phrase-extraction component 110 also identifies “congenitalhyperbilirubinemia”, which is an anchor term in subgraph 202B, asaligned with “congenital hyperbilirubinemia”, which is an anchor term insubgraph 204B. Entailment-phrase-extraction component 110 alsodetermines that although these two anchor terms are aligned, they bothappear in respective non-aligned portions of subgraph 202 and subgraph204. Entailment-phrase-extraction component 110 identifies a textfragment connecting “congenital hyperbilirubinemia” in non-alignedsubgraph 202B to the term “is” in aligned subgraph 202A, and identifiesan additional text fragment connecting “congenital hyperbilirubinemia”in non-aligned subgraph 204B to “is” in aligned subgraph 204A. Eachidentified text fragment becomes an element in the resulting entailmentpair: “<X> is cause of <Y>←<X> is explanation for <Y>”.

Referring now to FIGS. 1-3 , generally, a natural language processing(NLP) system, such as QA system 102, may receive and analyze multipletext pairs (for example, question-and-answer pairs); retrieve multiplepassages in response to queries generated based on the multipletext-pairs; and generate multiple entailment phrases. At one or morestages in this process, the NLP system may rank its results. Forexample, an entailment phrase may be scored, and ranked according tothat score. The score for an entailment phrase may be based on a varietyof factors, including, for example, one or more of the following: ascore of a passage from which a portion of the entailment phrase isextracted; the number of times the same entailment phrase is extractedusing different passages; the number of algorithms whose processing ofthe same QA pair and passage yield the same entailment phrase; and otherfactors.

Referring now to FIG. 4 , a schematic of an example of a cloud computingnode (which may be, for example, QA system 102 in FIG. 1 ) is shown.Cloud computing node 10 is only one example of a suitable cloudcomputing node and is not intended to suggest any limitation as to thescope of use or functionality of embodiments of the invention describedherein. Regardless, cloud computing node 10 is capable of beingimplemented and/or performing any of the functionality set forthhereinabove.

In cloud computing node 10 there is a computer system/server 12, whichis operational with numerous other general purpose or special purposecomputing system environments or configurations. Examples of well-knowncomputing systems, environments, and/or configurations that may besuitable for use with computer system/server 12 include, but are notlimited to, personal computer systems, server computer systems, thinclients, thick clients, hand-held or laptop devices, multiprocessorsystems, microprocessor-based systems, set top boxes, programmableconsumer electronics, network PCs, minicomputer systems, mainframecomputer systems, and distributed cloud computing environments thatinclude any of the above systems or devices, and the like.

Computer system/server 12 may be described in the general context ofcomputer system-executable instructions, such as program modules, beingexecuted by a computer system. Generally, program modules may includeroutines, programs, objects, components, logic, data structures, and soon that perform particular tasks or implement particular abstract datatypes. Computer system/server 12 may be practiced in distributed cloudcomputing environments where tasks are performed by remote processingdevices that are linked through a communications network. In adistributed cloud computing environment, program modules may be locatedin both local and remote computer system storage media including memorystorage devices.

As shown in FIG. 4 , computer system/server 12 in cloud computing node10 is shown in the form of a general-purpose computing device. Thecomponents of computer system/server 12 may include, but are not limitedto, one or more processors or processing units 16, a system memory 28,and a bus 18 that couples various system components including systemmemory 28 to processor 16.

Bus 18 represents one or more of any of several types of bus structures,including a memory bus or memory controller, a peripheral bus, anaccelerated graphics port, and a processor or local bus using any of avariety of bus architectures. By way of example, and not limitation,such architectures include Industry Standard Architecture (ISA) bus,Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, VideoElectronics Standards Association (VESA) local bus, and PeripheralComponent Interconnects (PCI) bus.

Computer system/server 12 typically includes a variety of computersystem readable media. Such media may be any available media that isaccessible by computer system/server 12, and it includes both volatileand non-volatile media, removable and non-removable media.

System memory 28 can include computer system readable media in the formof volatile memory, such as random access memory (RAM) 30 and/or cachememory 32.

Computer system/server 12 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 34 can be provided forreading from and writing to a non-removable, non-volatile magnetic media(not shown and typically called a “hard drive”). Although not shown, amagnetic disk drive for reading from and writing to a removable,non-volatile magnetic disk (e.g., a “floppy disk”), and an optical diskdrive for reading from or writing to a removable, non-volatile opticaldisk such as a CD-ROM, DVD-ROM or other optical media can be provided.In such instances, each can be connected to bus 18 by one or more datamedia interfaces. As will be further depicted and described below,memory 28 may include at least one program product having a set (e.g.,at least one) of program modules that are configured to carry out thefunctions of embodiments of the invention.

Program/utility 40, having a set (at least one) of program modules 42,may be stored in memory 28 by way of example, and not limitation, aswell as an operating system, one or more application programs, otherprogram modules, and program data. Each of the operating system, one ormore application programs, other program modules, and program data orsome combination thereof, may include an implementation of a networkingenvironment. Program modules 42 generally carry out the functions and/ormethodologies of embodiments of the invention as described herein.

Computer system/server 12 may also communicate with one or more externaldevices 14 such as a keyboard, a pointing device, a display 24, etc.;one or more devices that enable a user to interact with computersystem/server 12; and/or any devices (e.g., network card, modem, etc.)that enable computer system/server 12 to communicate with one or moreother computing devices. Such communication can occur via Input/Output(I/O) interfaces 22. Still yet, computer system/server 12 cancommunicate with one or more networks such as a local area network(LAN), a general wide area network (WAN), and/or a public network (e.g.,the Internet) via network adapter 20. As depicted, network adapter 20communicates with the other components of computer system/server 12 viabus 18. It should be understood that although not shown, other hardwareand/or software components could be used in conjunction with computersystem/server 12. Examples, include, but are not limited to: microcode,device drivers, redundant processing units, external disk drive arrays,RAID systems, tape drives, and data archival storage systems, etc.

Referring now to FIG. 5 , illustrative cloud computing environment 50 isdepicted. As shown, cloud computing environment 50 comprises one or morecloud computing nodes 10 with which local computing devices used bycloud consumers, such as, for example, personal digital assistant (PDA)or cellular telephone 54A, desktop computer 54B, laptop computer 54C,and/or automobile computer system 54N may communicate. Nodes 10 maycommunicate with one another. They may be grouped (not shown) physicallyor virtually, in one or more networks, such as Private, Community,Public, or Hybrid clouds as described hereinabove, or a combinationthereof. This allows cloud computing environment 50 to offerinfrastructure, platforms and/or software as services for which a cloudconsumer does not need to maintain resources on a local computingdevice. It is understood that the types of computing devices 54A-N shownin FIG. 5 are intended to be illustrative only and that cloud computingnodes 10 and cloud computing environment 50 can communicate with anytype of computerized device over any type of network and/or networkaddressable connection (e.g., using a web browser).

Referring now to FIG. 6 , a set of functional abstraction layersprovided by cloud computing environment 50 (FIG. 5 ) is shown. It shouldbe understood in advance that the components, layers, and functionsshown in FIG. 6 are intended to be illustrative only and embodiments ofthe invention are not limited thereto. As depicted, the following layersand corresponding functions are provided:

Hardware and software layer 60 includes hardware and softwarecomponents. Examples of hardware components include: mainframes 61; RISC(Reduced Instruction Set Computer) architecture based servers 62;servers 63; blade servers 64; storage devices 65; and networks andnetworking components 66. In some embodiments, software componentsinclude network application server software 67 and database software 68.

Virtualization layer 70 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers71; virtual storage 72; virtual networks 73, including virtual privatenetworks; virtual applications and operating systems 74; and virtualclients 75.

In one example, management layer 80 may provide the functions describedbelow. Resource provisioning 81 provides dynamic procurement ofcomputing resources and other resources that are utilized to performtasks within the cloud computing environment. Metering and Pricing 82provide cost tracking as resources are utilized within the cloudcomputing environment, and billing or invoicing for consumption of theseresources. In one example, these resources may comprise applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal 83 provides access to the cloud computing environment forconsumers and system administrators. Service level management 84provides cloud computing resource allocation and management such thatrequired service levels are met. Service Level Agreement (SLA) planningand fulfillment 85 provide pre-arrangement for, and procurement of,cloud computing resources for which a future requirement is anticipatedin accordance with an SLA.

Workloads layer 90 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation 91; software development and lifecycle management 92; virtualclassroom education delivery 93; data analytics processing 94;transaction processing 95; natural language processing 96, includingthose described in connection with FIGS. 1-3 , above.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

What is claimed is:
 1. A computer system for generating a textualentailment pair, comprising: one or more computer devices each havingone or more processors and one or more tangible storage devices; and aprogram embodied on at least one of the one or more storage devices, theprogram having a plurality of program instructions for execution by theone or more processors, the program instructions comprising instructionsfor: receiving first and second texts from an input source, wherein theinput source is a QA pipeline; querying a passage database using thefirst and second texts using an information retrieval engine in anatural language processing pipeline, wherein the first text is aquestion and the second text is a candidate answer; retrieving a passagefrom the passage database in response to the query using the informationretrieval engine in the natural language processing pipeline, whereinthe passage is retrieved based on a highest ranking within a ranked listof passages, wherein the ranked list is based on an expected degree ofrelevance of textual data to a domain of the question; selecting aplurality of anchor pairs in the first text and in the retrievedpassage, using a plurality of term matchers, wherein each of theplurality of term matchers uses a corresponding algorithm to identify amatch between a first anchor term in the first text and a second anchorterm in the passage retrieved, the anchor pair comprising two anchorterms each corresponding to a term representing an entity linked acrossat least two passages or two questions; generating an entailment pairbased on the selected anchor pairs, the entailment pair comprising apair of text fragments, wherein for terms of each of the plurality ofanchor pairs, (1) one term is from the question, and one term is fromthe passage; (2) the terms are both anchor terms; and (3) the terms havebeen matched to each other by the plurality of term matchers; andassigning a score to the generated entailment pair, wherein the scoreassigned a first time the entailment pair is generated is an initialscore; retrieving a plurality of additional passages in response to atleast one additional query; performing term-matching between terms inthe question and terms in the plurality of additional passages; scoringthe plurality of additional passages retrieved based on theterm-matching between the terms in the question and the terms in theplurality of additional passages; ranking the plurality of additionalpassages retrieved based on the score of the plurality of additionalpassages; generating the entailment pair at least one additional timebased on the plurality of additional passages retrieved; adjusting thescore of the generated entailment pair each additional time theentailment pair is generated from the plurality of additional passages,wherein the score is adjusted using a weighting multiplier based on theranking of the at least one additional passage from which the generatedentailment pair is extracted; and storing the generated entailment pairin an entailment database if the score exceeds a threshold value.
 2. Thesystem of claim 1, wherein the program instructions further compriseinstructions for: retrieving a plurality of additional passages;performing term-matching between terms in the question and terms in theretrieved passage and in the plurality of additional passages; andscoring the retrieved passage and the plurality of additional passagesbased on the term matching, and wherein generating an entailment pair isbased on the scoring.
 3. The system of claim 1, wherein a passage scorerperforms term matching between at least one term in the first text andat least one term in the retrieved passage, and the program instructionsfurther comprise instructions for: generating one or more parse treesfor the first text and the retrieved passage; identifying one or morefocus terms in respective parse trees of the first text and theretrieved passage; identifying one or more focus subgraphs for the firsttext and for the retrieved passage based on the identified focus terms;and aligning at least one focus subgraph of the first text with at leastone subgraph of the retrieved passage.
 4. The system of claim 1, whereinthe program instructions further comprise instructions for: identifyinga first anchor term in the first text matched by a term matcher to asecond anchor term in the retrieved passage; and designating the firstanchor term and the second anchor term as an anchor pair.
 5. The systemof claim 1, wherein the program instructions further compriseinstructions for: aligning a first text fragment, in the first text,with a second text fragment, in the retrieved passage, based on alignedfirst and second anchor terms in an first anchor pair, wherein the firstanchor term appears in the first text fragment, and the second anchorterm appears in the second text fragment; aligning a third anchor termin the first text with a fourth anchor term in the retrieved passage,wherein the third anchor term is not in the firs text fragment and thefourth anchor term is not in the second text fragment; identifying athird text fragment, in the first text, connecting the third anchor termto the first text fragment; and identifying a fourth text fragment, inthe retrieved passage, connecting the fourth anchor term to the secondtext fragment, wherein the entailment pair comprises the third andfourth text fragments.
 6. A computer program product for generating atextual entailment pair by an electronic natural language processing(NLP) system, comprising a non-transitory tangible storage device havingprogram code embodied therewith, the program code executable by aprocessor of a computer to perform a method, the method comprising:receiving first and second texts from an input source, by the NLPsystem, each corresponding to a natural language text; querying, by theNLP system, a passage database using the first and second texts using aninformation retrieval engine in a natural language processing pipeline,wherein the first text is a question and the second text is a candidateanswer; retrieving, by the NLP system, a passage from the passagedatabase in response to the query using the information retrieval enginein the natural language processing pipeline, wherein the passage isretrieved based on a highest ranking within a ranked list of passages,wherein the ranked list is based on an expected degree of relevance oftextual data to a domain of the question; selecting, by the NLP system,a plurality of anchor pairs in the first text and in the retrievedpassage, using a plurality of term matchers, wherein each of theplurality of term matchers uses a corresponding algorithm to identify amatch between a first anchor term in the first text and a second anchorterm in the passage retrieved, the anchor pair comprising two anchorterms each corresponding to a term representing an entity linked acrossat least two passages or two questions; generating, by the NLP system,an entailment pair based on the selected anchor pairs, the entailmentpair comprising a pair of text fragments, wherein for terms of each ofthe plurality of anchor pairs, (1) one term is from the question, andone term is from the passage; (2) the terms are both anchor terms; and(3) the terms have been matched to each other by the plurality of termmatchers; and assigning, by the NLP system, a score to the generatedentailment pair, wherein the score assigned a first time the entailmentpair is generated is an initial score; retrieving, by the NLP system, aplurality of additional passages in response to at least one additionalquery; performing, by the NLP system, term-matching between terms in thequestion and terms in the plurality of additional passages; scoring, bythe NLP system, the plurality of additional passages retrieved based onterm-matching between the terms in the question and the terms in theplurality of additional passages; ranking, by the NLP system, theplurality of additional passages retrieved based on the score of theplurality of additional passages; generating, by the NLP system, theentailment pair at least one additional time based on the plurality ofadditional passages retrieved; adjusting, by the NLP system, the scoreof the generated entailment pair each additional time the entailmentpair is generated from the plurality of additional passages, wherein thescore is adjusted using a weighting multiplier based on the ranking ofthe at least one additional passage from which the generated entailmentpair is extracted; and storing, by the NLP system, the generatedentailment pair in an entailment database if the score exceeds athreshold value.
 7. The system of claim 1, further comprising:generating a list of passages, wherein the list of passages is comprisedof one or more passages from which at least one entailment pair isextracted; ranking the one or more passages within the list of passages;and adjusting the score of the generated entailment pair using aweighting multiplier based on the ranking of the passage in which thegenerated entailment pair is extracted.
 8. The system of claim 7,wherein the weighting multiplier is based on an average passage score oftwo or more passages from which the generated entailment pair isextracted.
 9. The system of claim 3, wherein aligning the at least onefocus subgraph of the first text with the at least one subgraph of theretrieved passage is based on a plurality of term alignment scores ofthe at least one focus subgraph.
 10. The system of claim 9, wherein theplurality of term alignment scores of the at least one focus subgraphare determined using a decaying sum of the plurality of term alignmentscores in the at least one focus subgraph.
 11. A method for generating atextual entailment pair by an electronic natural language processingsystem, comprising: receiving first and second texts from an inputsource, wherein the input source is a QA pipeline; querying a passagedatabase using the first and second texts using an information retrievalengine in a natural language processing pipeline, wherein the first textis a question and the second text is a candidate answer; retrieving apassage from the passage database in response to the query using theinformation retrieval engine in the natural language processingpipeline, wherein the passage is retrieved based on a highest rankingwithin a ranked list of passages, wherein the ranked list is based on anexpected degree of relevance of textual data to a domain of thequestion; selecting a plurality of anchor pairs in the first text and inthe retrieved passage, using a plurality of term matchers, wherein eachof the plurality of term matchers uses a corresponding algorithm toidentify a match between a first anchor term in the first text and asecond anchor term in the passage retrieved, the anchor pair comprisingtwo anchor terms each corresponding to a term representing an entitylinked across at least two passages or two questions; generating anentailment pair based on the selected anchor pairs, the entailment paircomprising a pair of text fragments, wherein for terms of each of theplurality of anchor pairs, (1) one term is from the question, and oneterm is from the passage; (2) the terms are both anchor terms; and (3)the terms have been matched to each other by the plurality of termmatchers; and assigning a score to the generated entailment pair,wherein the score assigned a first time the entailment pair is generatedis an initial score; retrieving a plurality of additional passages inresponse to at least one additional query; performing term-matchingbetween terms in the question and terms in the plurality of additionalpassages; scoring the plurality of additional passages retrieved basedon the term-matching between the terms in the question and the terms inthe plurality of additional passages; ranking the plurality ofadditional passages retrieved based on the score of the plurality ofadditional passages; generating the entailment pair at least oneadditional time based on the plurality of additional passages retrieved;adjusting the score of the generated entailment pair each additionaltime the entailment pair is generated from the plurality of additionalpassages, wherein the score is adjusted using a weighting multiplierbased on the ranking of the at least one additional passage from whichthe generated entailment pair is extracted; and storing the generatedentailment pair in an entailment database if the score exceeds athreshold value.
 12. The system of claim 1, wherein each text fragmentwithin the pair of text fragments is a node, wherein each node of thepair of text fragments connecting the anchor terms is a shortest path toan aligned focus subgraph.
 13. The computer program product of claim 6,wherein each text fragment within the pair of text fragments is a node,wherein each node of the pair of text fragments connecting the anchorterms is a shortest path to an aligned focus subgraph.
 14. The method ofclaim 11, wherein each text fragment within the pair of text fragmentsis a node, wherein each node of the pair of text fragments connectingthe anchor terms is a shortest path to an aligned focus subgraph. 15.The system of claim 1, wherein each of the plurality of additionalpassages retrieved includes the pair of text fragments connecting theanchor terms.
 16. The computer program product of claim 6, wherein eachof the plurality of additional passages retrieved includes the pair oftext fragments connecting the anchor terms.
 17. The method of claim 11,wherein each of the plurality of additional passages retrieved includesthe pair of text fragments connecting the anchor terms.
 18. The systemof claim 1, wherein the program instructions further compriseinstructions for: retrieving the generated entailment pair and at leastone other entailment pair from the entailment pair knowledgebase,wherein both the generated entailment pair and the at least one otherentailment pair exceed the threshold value; and providing the generatedentailment pair and the at least one other entailment pair to a processin the question-answering (QA) pipeline.
 19. The computer programproduct of claim 6, further comprising: retrieving, by the NLP system,the generated entailment pair and at least one other entailment pairfrom the entailment pair knowledgebase, wherein both the generatedentailment pair and the at least one other entailment pair exceed thethreshold value; and providing, by the NLP system, the generatedentailment pair and the at least one other entailment pair to a processin the question-answering (QA) pipeline.